
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
  <head>
    <title>XMerL - XML processing tools for Erlang</title>
  </head>

  <body 
    TEXT="#000000" 
    BGCOLOR="#FFFFFF" 
    LINK="#0000FF" 
    VLINK="#FF0000" 
    ALINK="#FF0000">

<h1>1. XMerL - XML processing tools for Erlang</h1>


<h2>1.1 xmerl_scan - the XML processor</h2>

<p>The (non-validating) XML processor is 
activated through <code>xmerl_scan:string/[1,2]</code> or 
 <code>xmerl_scan:file/[1,2]</code>.
It returns records of the type defined in xmerl.hrl.</p>

<p>As far as I can tell, xmerl_scan implements the complete XML 1.0 spec,
including:</p>
<ul>
 <li>entity expansion</li>
 <li>fetching and parsing external DTDs</li>
 <li>contitional processing</li>
 <li>UniCode</li>
 <li>XML Names</li>
</ul>

xmerl_scan:string(Text [ , Options ]) -> #xmlElement{}.
xmerl_scan:file(Filename [ , Options ]) -> #xmlElement{}.

<p>The Options are basically to specify the behaviour of the scanner. 
See the source code for details, but you can specify funs to handle 
scanner events (event_fun), process the document entities once identified 
(hook_fun), and decide what to do if the scanner runs into eof before 
the document is complete (continuation_fun).</p>
<p>You can also specify a path (fetch_path) as a list of directories to
search when fetching files. If the file in question is not in the fetch_path,
the URI will be used as a file name.</p>

  <h3>1.1.1 Customization functions</h3>

<p>The XML processor offers a number of hooks for customization. 
These hooks are defined as function objects, and can be provided by 
the caller.</p>

<p>The following customization functions are available. If they also have
access to their own state variable, the access function for this state
is identified within parentheses:</p>
    <ul>
      <li>event function (<code>xmerl_scan:event_state/[1,2]</code>)</li>
      <li>hook function (<code>xmerl_scan:hook_state/[1,2]</code>)</li>
      <li>fetch function (<code>xmerl_scan:fetch_state/[1,2]</code>)</li>
      <li>continuation function (<code>xmerl_scan:cont_state/[1,2]</code>)</li>
      <li>rules function (<code>xmerl_scan:rules_state/[1,2]</code>)</li>
      <li>accumulator function</li>
      <li>close function</li>
    </ul>
<p>For all of the above state access functions, the function with one 
argument (e.g. <code>event_fun(GlobalState)</code>) will read the state 
variable, while the function with two arguments (e.g.:
<code>event_fun(NewStateData, GlobalState)</code>) will modify it.</p>

<p>For each function, the description starts with the syntax for specifying
the function in the <code>Options</code> list. The general forms are
<code>{Tag, Fun}</code>, or <code>{Tag, Fun, LocalState}</code>. The 
second form can be used to initialize the state variable in question.</p>

    <h4>1.1.1.0 User State</h4>

<p>All customization functions are free to access a &quot;User state&quot; 
variable. Care must of course be taken to coordinate the use of this
state. It is recommended that functions, which do not really have anything
to contribute to the &quot;global&quot; user state, use their own state
variable instead. Another option (used in e.g. <code>xmerl_eventp.erl</code>)
is for customization functions to share one of the local states (in
<code>xmerl_eventp.erl</code>, the continuation function and the fetch
function both acces the <code>cont_state</code>.)</p>

<p>Functions to access user state:</p>

    <ul>
      <li><code>xmerl_scan:user_state(GlobalState)</code></li>
      <li><code>xmerl_scan:user_state(UserState', GlobalState)</code></li>
    </ul>

    <h4>1.1.1.1 Event Function</h4>

<code>{event_fun, fun()} | {event_fun, fun(), LocalState}</code>

<p>The event function is called at the beginning and at the end of a 
parsed entity. It has the following format and semantics:</p>

<pre>
fun(Event, GlobalState) ->
   EventState = xmerl_scan:event_state(GlobalState),
   EventState' = foo(Event, EventState),
   GlobalState' = xmerl_scan:event_state(EventState', GlobalState)
end.
</pre>

  <h4>1.1.1.2 Hook Function</h4>

<code>{hook_fun, fun()} | {hook_fun, fun(), LocalState}</code>

<p>The hook function is called when the processor has parsed a complete
entity. Format and semantics:</p>

<pre>
fun(Entity, GlobalState) ->
   HookState = xmerl_scan:hook_state(GlobalState),
   {TransformedEntity, HookState'} = foo(Entity, HookState),
   GlobalState' = xmerl_scan:hook_state(HookState', GlobalState),
   {TransformedEntity, GlobalState'}
end.
</pre>

<p>The relationship between the event function, the hook function and the
accumulator function is as follows:</p>
    <ol>
      <li>The event function is first called with an 'ended' event for
	the parsed entity.</li>
      <li>The hook function is called, possibly re-formatting the entity.</li>
      <li>The acc function is called in order to (optionally) add the 
	re-formatted entity to the contents of its parent element.</li>
    </ol>


  <h4>1.1.1.3 Fetch Function</h4>

<code>{fetch_fun, fun()} | {fetch_fun, fun(), LocalState}</code>

<p>The fetch function is called in order to fetch an external resource
(e.g. a DTD).</p>

<p>The fetch function can respond with three different return values:</p>

    <pre>
    Result ::=
      {ok, GlobalState'} |
      {ok, {file, Filename}, GlobalState'} |
      {ok, {string, String}, GlobalState'}
    </pre>

<p>Format and semantics:</p>

<pre>
fun(URI, GlobalState) ->
   FetchState = xmerl_scan:fetch_state(GlobalState),
   Result = foo(URI, FetchState).  % Result being one of the above
end.
</pre>


    <h4>1.1.1.4 Continuation Function</h4>

<code>{continuation_fun, fun()} | {continuation_fun, fun(), LocalState}</code>

<p>The continuation function is called when the parser encounters the end
of the byte stream. Format and semantics:</p>


<pre>
fun(Continue, Exception, GlobalState) ->
   ContState = xmerl_scan:cont_state(GlobalState),
   {Result, ContState'} = get_more_bytes(ContState),
   GlobalState' = xmerl_scan:cont_state(ContState', GlobalState),
   case Result of
      [] ->
         GlobalState' = xmerl_scan:cont_state(ContState', GlobalState),
         Exception(GlobalState');
      MoreBytes ->
         {MoreBytes', Rest} = end_on_whitespace_char(MoreBytes),
         ContState'' = update_cont_state(Rest, ContState'),
         GlobalState' = xmerl_scan:cont_state(ContState'', GlobalState),
         Continue(MoreBytes', GlobalState')
   end
end.
</pre>

    <h4>1.1.1.5 Rules Functions</h4>

<pre>
{rules, ReadFun : fun(), WriteFun : fun(), LocalState} |
{rules, Table : ets()}</pre>

<p>The rules functions take care of storing scanner information in a rules
database. User-provided rules functions may opt to store the information
in mnesia, or perhaps in the user_state(LocalState).</p>
<p>The following modes exist:</p>

    <ul>
      <li>If the user doesn't specify an option, the scanner creates an
	ets table, and uses built-in functions to read and write data to it.
	When the scanner is done, the ets table is deleted.</li>
      <li>If the user specifies an ets table via the 
	<code>{rules, Table}</code> option, the scanner uses this table.
	When the scanner is done, it does <strong>not</strong> delete the
	table.</li>
      <li>If the user specifies read and write functions, the scanner will
	use them instead.</li>
    </ul>

<p>The format for the read and write functions are as follows:</p>

<pre>
WriteFun(Context, Name, Definition, ScannerState) -> NewScannerState.
ReadFun(Context, Name, ScannerState) -> Definition | undefined.
</pre>

<p>Here is a summary of the data objects currently being written by the
scanner:</p>

<p><table border=1>
	<tr>
	  <th>Context</th>
	  <th>Key Value</th>
	  <th>Definition</th>
	</tr>
	<tr>
	  <td>notation</td>
	  <td>NotationName</td>
	  <td><code>{system, SL} | {public, PIDL, SL}</code></td>
	</tr>
	<tr>
	  <td>elem_def</td>
	  <td>ElementName</td>
	  <td><code>#xmlElement{content = ContentSpec}<code></td>
	</tr>
	<tr>
	  <td>parameter_entity</td>
	  <td>PEName</td>
	  <td><code>PEDef<code></td>
	</tr>
	<tr>
	  <td>entity</td>
	  <td>EntityName</td>
	  <td><code>EntityDef<code></td>
	</tr>
      </table>
</p>

<p><pre>
ContentSpec ::= empty | any | ElemContent
ElemContent ::= {Mode, Elems}
Mode        ::= seq | choice
Elems       ::= [Elem]
Elem        ::= '#PCDATA' | Name | ElemContent | {Occurrence, Elems}
Occurrence  ::= '*' | '?' | '+'
</pre></p>
<p>Note: When &lt;Elem&gt; is not wrapped with &lt;Occurrence&gt;, 
(Occurrence = once) is implied.</p>


    <h4>1.1.1.5 Accumulator Function</h4>

<code>{acc_fun, fun()} | {acc_fun, fun(), LocalState}</code>

<p>The accumulator function is called to accumulate the contents of an entity.
When parsing very large files, it may not be desireable to do so.
In this case, an acc function can be provided that simply doesn't
accumulate.</p>

<p>Note that it is possible to even modify the parsed entity before
accumulating it, but this must be done with care. <code>xmerl_scan</code>
performs post-processing of the element for namespace management. Thus,
the element must keep its original structure for this to work.</p>

<p>The acc function has the following format and semantics:</p>

<pre>
%% default accumulating acc fun
fun(ParsedEntity, Acc, GlobalState) ->
   {[X|Acc], GlobalState}.

%% non-accumulating acc fun
fun(ParsedEntity, Acc, GlobalState) ->
   {Acc, GlobalState}.
</pre>

  <h4>1.1.1.5 Close Function</h4>

<p>The close function is called when a document (either the main document
or an external DTD) has been completely parsed. When xmerl_scan was started
using <code>xmerl_scan:file/[1,2]</code>, the file will be read in full, and
closed immediately, before the parsing starts, so when the close function is
called, it will not need to actually close the file. In this case, the close
function will be a good place to modify the state variables.</p>

<p>Format and semantics:</p>

<pre>
fun(GlobalState) ->
   GlobalState' = ....  % state variables may be altered
</pre>



<h2>2. XPATH</h2>

<code>
xmerl_xpath:string(QueryString, #xmlElement{}) ->
	[DocEntity]

DocEntity :	#xmlElement{} 
		| #xmlAttribute{} 
		| #xmlText{} 
		| #xmlPI{}
		| #xmlComment{}
</code>

<p>The xmerl_xpath module does seem to handle the entire XPATH 1.0 spec,
but I haven't tested that much yet. The grammar is defined in 
<code>xmerl_xpath_parse.yrl</code>. 
The core functions are defined in <code>xmerl_xpath_pred.erl</code>.
</p>

<h3>2.1 Some useful shell commands for debugging the XPath parser</h3>
<code>
c(xmerl_xpath_scan).
yecc:yecc("xmerl_xpath_parse.yrl", "xmerl_xpath_parse", true, []).
c(xmerl_xpath_parse).

xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("position() > -1")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("5 * 6 div 2")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("5 + 6 mod 2")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("5 * 6")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("5 * 6")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("-----6")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("parent::node()")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("descendant-or-self::node()")).
xmerl_xpath_parse:parse(xmerl_xpath_scan:tokens("parent::processing-instruction('foo')")).
</code>


<h2>3. Erlang Data Structure Export</h2>

<p>The idea as follows:</p>

<p>The Erlang data structure should look like this:

<code>
Element:	{Tag, Attributes, Content}
Tag :		atom()
Attributes:	[{Key, Value}]
Content:	String | [String | Element]
String:		[byte()] | binary()
			% anything that goes through list_to_binary/1
</code>

<p>Some short forms are allowed:</p>

<code>
{Tag, Content}	-> {Tag, [], Content}
Tag		-> {Tag, [], []}
</code>


<p><code>xmerl:export_simple(Data, Callback)</code> takes the above 
data structure and exports it, using the callback module 
<code>Callback</code>.</p>

<p>The callback module should contain hook functions for all tags 
present in the data structure. The hook function must have the format:</p>

<code>
Tag(Data, Attrs, Parents, E)
</code>

<p>where E is an <code>#xmlElement{}</code> record 
(see <code>xmerl.hrl</code>).</p>
<p>Attrs is converted from the simple <code>[{Key, Value}]</code> 
to <code>[#xmlAttribute{}]</code></p>
<p>Parents is a list of <code>[{ParentTag, ParentTagPosition}]</code>.</p>

<p>The hook function should return either the Data to be exported, or 
the tuple <code>{'#xml-redefine#', NewStructure}</code> 
(see <code>xmerl_html:'tuple.list'/4</code>).</p>


<p>The callback module can inherit definitions from other callback modules,
through the required function 
<code>'#xml-interitance#() -> [ModuleName]</code>.

<p>As long as a tag is represented in one of the callback modules, 
things will work. It is of course also possible to redefine a tag.</p>


